[Job Debugging] Basic Implement Of Job Debugging. #2272

ydye · 2019-03-05T08:53:47Z

This PR will achieve

The ability to hold the failed job which is casued by user's error. User error means there is something wrong happened in user's command and the command exit with a none-zero code.

This PR won't solve

We can't hold the failed job which is caused by OpenPAI's system error. System error means the job is killed by system alerts or system issue. For example, user's job will be killed if the container write too much data into local path.

Relative Issues:

Design Overview: Better Job Debugging Experience for End Users #2210
- Reserve job container whose user command exits with none zero code. [Job Debugging] Container reservation when the job failed due to users' error. #2213
- An option for user to enable job debugging feature for their job. [Job Debugging] An option for user to enable debugging reservation #2214
- Tutorial for user to use job debugging. [Job Debugging] Tutorial for user to enable job debugging #2259
- Configurable reservation time. [Job Debugging] Make reservation time configurable to admin in cluster-configuration #2252

Sub-PR detail information

Tutorial for user Tutorial for user to use job debugging. #2264

Provide the document for user to use job debugging feature

Configurable job debugging reservation time for admin Make debugging reservation time configurable. #2257

Admin could configure the debugging reservation time in services-configuration.yaml -> rest-server -> debugging-reservation-seconds. The default value will be 604800 seconds.

Debug option for user and the ability to reserved a container due to user error [Job Debugging] If user command's exitcode is none zero, container will be reserved. #2225

When user submit the job, set the following property in the jobEnv of jobConfig. If the job's user command fails, the container will be kept for 1 weeks. And user could debug the container after ssh to it. After debugging, user should manually stop it to recycle the system resources.

When submitting job from webportal:

when submitting job from json file:

  "jobEnvs": {
    "isDebug": true
  }

When you job failed, click into the `Go to Tracking Page` and looking into the `stdout`. You could find following log.

[INFO] USER COMMAND START

job has finished with exit code 2
=============================================================================
======   The job container failed, so it will be reserved for 1 week   ======
======          After debugging, please stop the job manually.         ======
=============================================================================

Future Plan

Provide more detailed failure info, if the job is killed due to system error. [Job Debugging] Provide detail information when the job container exits. #2218
Policy to collect the job status such as job debugging on each node. [Job Debugging] Approach to collect the information of container in debugging reservation. #2215
Show the job which is reserved in job debugging status for user. [Job Debugging] Display the debugging reservation status for job in webportal #2216
The ability to email user when the job is reserved for job debugging. [Job Debugging] Approach to notify users when their jobs are in debugging reservation #2217

…ll be reserved. (#2225)

…ob debugging (#2248)

coveralls · 2019-03-05T09:01:17Z

Coverage increased (+0.02%) to 52.643% when pulling 3881b3c on job-debugging-dev into 8cb09a1 on master.

coveralls · 2019-03-05T09:01:19Z

Coverage increased (+0.02%) to 52.643% when pulling 3881b3c on job-debugging-dev into 8cb09a1 on master.

coveralls · 2019-03-05T09:01:20Z

Coverage increased (+0.02%) to 52.649% when pulling ab49e4d on job-debugging-dev into 8cb09a1 on master.

src/rest-server/src/config/paiConfig.js

* add a dashboard in grafana to list all tasks in node (#2197) * Fix format in issue templates (#2233) Fix format in issue templates: - remove trailing spaces - change chinese colon into english * Fix auto retries when out of memory. (#1108) * Distinguish cgroup OOM from dmesg. * Remove cgroup OOM detection Make all OOM cause exiting by 5 * Exit 55 when OOM * Refine homepage for new users (#2155) Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform. * Fix yarn container failed when docker container exited quickly. (#2256) * REST server: remove expires in JWT payload of unit test (#2263) * Deploy: add explicit config field in webportal plugin (#2251) * Deploy: add explicit config field in webportal plugin * Fix json.dumps * t * fix * Update PLUGINS.md * Update webportal.md * alert on unhealthy gpu (#2209) * Pylon: fix double start query in yarn redirect (#2258) * Pylon: fix double start query in yarn redirect * Hide debug info in docker-compose.yaml * adapt user transfer script to new config (#2266) * Webportal: add pai-version attribute to <pai-plugin> (#2245) * Webportal: add pai-version attribute to <pai-plugin> * Use preprocess to apply window.PAI_VERSION * set version in layout.html * Fix ib drivers bug (#2269) * FIx ib installation script bug (#2271) * [BUG] Fix hadoop ai build path (#2262) * fix hadoop ai build bugs * refine * Web portal submit job: support init json from sessionStorage. (#2253) * YARN and HDFS log persistence (#2244) * rm log persist * change log dir to host * persist nm log to host * resolve conflict * persist namenode log * persist data node log * add comments * move log path to common pai storage * use twisted in yarn-exporter (#2273) * [Job Debugging] Basic Implement Of Job Debugging. (#2272) * Refine document for new user to submit job (#2278) 1. add new guidance to submit job for beginners. 2. refine homepage to connect with new guidance. 3. reorganize content of troubleshooting for next refactoring. 4. fix links in md files. * [Drivers] Fix the issue when installing IB drivers. (#2275) * fix can not report zombie process using gpu error (#2279) * fix external process error * add debug log * fix short ID and long ID do not match * use time based atomic ref to exchange info between threads * add test case for AtomicRef * fix bug in file remove (#2288) * fix hadoop build error (#2296) * export vc/node related metrics from yarn (#2289) * 720 * open hdfs explorer in view container enable tslint rule "ordered-imports" * add tslint rule for indent * add home button to hdfs explorer's navigation; adjust octicon's color * fix lint error * [VS Code] Add job list (#2160) * add job list view to pai extension * [VS Code] joblist fix (#2185) * eager load recent jobs when job submitted * avoid eager getChildren, and let vscode treeview.reveal do it implicitly * fix lint error

* add installation guide for VS code extension (#2223) * add installation guide for VS code extension * [VS Code] view container (#2301) * add a dashboard in grafana to list all tasks in node (#2197) * Fix format in issue templates (#2233) Fix format in issue templates: - remove trailing spaces - change chinese colon into english * Fix auto retries when out of memory. (#1108) * Distinguish cgroup OOM from dmesg. * Remove cgroup OOM detection Make all OOM cause exiting by 5 * Exit 55 when OOM * Refine homepage for new users (#2155) Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform. * Fix yarn container failed when docker container exited quickly. (#2256) * REST server: remove expires in JWT payload of unit test (#2263) * Deploy: add explicit config field in webportal plugin (#2251) * Deploy: add explicit config field in webportal plugin * Fix json.dumps * t * fix * Update PLUGINS.md * Update webportal.md * alert on unhealthy gpu (#2209) * Pylon: fix double start query in yarn redirect (#2258) * Pylon: fix double start query in yarn redirect * Hide debug info in docker-compose.yaml * adapt user transfer script to new config (#2266) * Webportal: add pai-version attribute to <pai-plugin> (#2245) * Webportal: add pai-version attribute to <pai-plugin> * Use preprocess to apply window.PAI_VERSION * set version in layout.html * Fix ib drivers bug (#2269) * FIx ib installation script bug (#2271) * [BUG] Fix hadoop ai build path (#2262) * fix hadoop ai build bugs * refine * Web portal submit job: support init json from sessionStorage. (#2253) * YARN and HDFS log persistence (#2244) * rm log persist * change log dir to host * persist nm log to host * resolve conflict * persist namenode log * persist data node log * add comments * move log path to common pai storage * use twisted in yarn-exporter (#2273) * [Job Debugging] Basic Implement Of Job Debugging. (#2272) * Refine document for new user to submit job (#2278) 1. add new guidance to submit job for beginners. 2. refine homepage to connect with new guidance. 3. reorganize content of troubleshooting for next refactoring. 4. fix links in md files. * [Drivers] Fix the issue when installing IB drivers. (#2275) * fix can not report zombie process using gpu error (#2279) * fix external process error * add debug log * fix short ID and long ID do not match * use time based atomic ref to exchange info between threads * add test case for AtomicRef * fix bug in file remove (#2288) * fix hadoop build error (#2296) * export vc/node related metrics from yarn (#2289) * 720 * open hdfs explorer in view container enable tslint rule "ordered-imports" * add tslint rule for indent * add home button to hdfs explorer's navigation; adjust octicon's color * fix lint error * [VS Code] Add job list (#2160) * add job list view to pai extension * [VS Code] joblist fix (#2185) * eager load recent jobs when job submitted * avoid eager getChildren, and let vscode treeview.reveal do it implicitly * fix lint error * [VS Code] default to generate jsonc job config file (#2368) * 720 * open hdfs explorer in view container enable tslint rule "ordered-imports" * add tslint rule for indent * add home button to hdfs explorer's navigation; adjust octicon's color * fix lint error * [VS Code] Add job list (#2160) * add job list view to pai extension * [VS Code] joblist fix (#2185) * eager load recent jobs when job submitted * avoid eager getChildren, and let vscode treeview.reveal do it implicitly * default to generate jsonc job config file * [VS Code] Refine error messages; Fix Cluster Explorer's bug * [VS Code] changelog and readme (#2429) * 539 * 712 * 452 * [VS Code] v0.11 compatible issue (#2457) * 536 * 600 * [VS Code] fix cluster explorer's right-click menu (#2463)

ydye and others added 7 commits February 28, 2019 14:30

[Job Debugging] If user command's exitcode is none zero, container wi…

7410b3d

…ll be reserved. (#2225)

[Job Debugging] Fix Issue that user can't stop the job in status of j…

0464c2c

…ob debugging (#2248)

Merge branch 'master' into job-debugging-dev

c2502ed

Merge branch 'master' into job-debugging-dev

1a0036e

Make debugging reservation time configurable. (#2257)

fd6039c

Tutorial for user to use job debugging. (#2264)

df287ae

solve conflict

3881b3c

ydye requested review from fanyangCS, sterowang, scarlett2018, mzmssg, yqwang-ms and Gerhut March 5, 2019 09:24

yqwang-ms approved these changes Mar 5, 2019

View reviewed changes

Gerhut reviewed Mar 5, 2019

View reviewed changes

src/rest-server/src/config/paiConfig.js Outdated Show resolved Hide resolved

ydye added 2 commits March 6, 2019 09:54

Update logger type.

5a86f37

bug fix

ab49e4d

ydye merged commit a236afa into master Mar 7, 2019

ydye deleted the job-debugging-dev branch April 3, 2019 07:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Job Debugging] Basic Implement Of Job Debugging. #2272

[Job Debugging] Basic Implement Of Job Debugging. #2272

ydye commented Mar 5, 2019 •

edited

Loading

coveralls commented Mar 5, 2019

coveralls commented Mar 5, 2019

coveralls commented Mar 5, 2019 •

edited

Loading

[Job Debugging] Basic Implement Of Job Debugging. #2272

[Job Debugging] Basic Implement Of Job Debugging. #2272

Conversation

ydye commented Mar 5, 2019 • edited Loading

This PR will achieve

This PR won't solve

Relative Issues:

Sub-PR detail information

When submitting job from webportal:

when submitting job from json file:

When you job failed, click into the Go to Tracking Page and looking into the stdout. You could find following log.

Future Plan

coveralls commented Mar 5, 2019

coveralls commented Mar 5, 2019

coveralls commented Mar 5, 2019 • edited Loading

ydye commented Mar 5, 2019 •

edited

Loading

When you job failed, click into the `Go to Tracking Page` and looking into the `stdout`. You could find following log.

coveralls commented Mar 5, 2019 •

edited

Loading